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estimate this noise. To this end, we simulate certain time series and random matrices 
which can model financial correlations. With our approach, different correlation 
structures buried under this noise can be detected. Moreover, we introduce a measure 
for the relation between noise and correlations. Our method is based on a power 
mapping which efficiently suppresses the noise. Neither further data processing nor 
additional input is needed. 
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1. Introduction 

There are different kinds of risk in a stock portfolio, for example, a systematic one due 
to the general trends in the economy affecting the entire market, and an unsystematic 
one due to events affecting only segments of the whole market under consideration, 
such as the companies within the same industrial branch or in one country JTJj. All this 
is borne out in the time series of the stocks. In a more general framework, the term 
"stock of a company" should be replaced by "risk factor" . The corresponding portfolios 
of a bank or an investment company can contain hundreds or thousands of financial 
instruments, depending on many different risk factors |3J. However, to be explicit, we 
will use the term "stocks" instead of "risk factors". Banks or portfolio managers are 
particularly interested in the unsystematic correlations 0, [|. We refer to them simply 
as correlations from now on. The elements of the correlation matrix are given through 
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the time average over a product of the properly normalized time series for two stocks. 
The number of companies considered defines the size of this correlation matrix. We will 
assume that this number is large enough for a statistical analysis. 

In recent years, it was shown that the true correlations one is interested in can 
be buried under noise in the commonly used correlation matrices: Laloux et al. [|[ 
studied an empirical correlation matrix and found that the bulk part of its spectral 
density can be modeled by a purely random matrix. Plerou et al. || worked out 
spectral fluctuations and also found that they are compatible with random matrix 
statistics. Random matrices are often used statistical models for quantum chaotic or 
related spectral problems, for reviews see Refs. || ff], ||. The presence of random matrix 
features was coined noise dressing of financial correlations. Burda et al. @ designed a 
proper model involving random Levy matrices. In practice, such noise can appear if the 
length of the time series employed to construct the correlation matrices is too short. 
Gopikrishnan et al. [JTOJ developed a method to reduce the noise by taking advantage of 
the fact that large eigenvalues outside the bulk of the spectral density can be associated 
with industrial branches. These authors [GIT] used this for portfolio optimization, see 



also Ref. [11|. The identification of branches through stock correlations was also studied 



by Mantegna [12 



Here, we present a new approach to identify and estimate the noise and, thereby, 
also the strength of the true correlations. As we do not use the large eigenvalues outside 
the bulk of the spectral density, our method is an alternative and a complement to the 
approach of Ref. |10[. It will be particularly useful if some of the large eigenvalues are 
not large enough to lie outside the bulk of the spectral density. Our method is based on 
a power mapping. It does not require any further data processing or any other type of 
input. To the best of our knowledge, it seems to be a new technique for the analysis of 
correlation matrices in time series problems. We have three goals: first, we develop an 
alternative method for noise identification, second, we introduce the power mapping as 
a new mathematical-statistical technique and discuss its features in some detail, third, 
we present our findings in a self-contained way to make possible a transfer to other time 
series problems and the corresponding correlation matrices. 

The article is organized as follows. In Sec. [2|, we outline the stochastic model, 
including a review of correlation matrices and a discussion of the noise dressing 
phenomenon. We study the dependence of the spectral density as a function of the 
length of the time series in Sec. |3|. In Sec. |], we introduce the power mapping and use it 
to identify and estimate the noise. We summarize and conclude in Sec. ||. Three more 
detailed discussions are given in the appendix. 



2. Stochastic Model 



We formulate the model for financial correlations in Sec. [2.1j Since we also aim at a 
heuristic analytical treatment later on, we do that in some detail. Correlation matrices 
and noise dressing are discussed in Sees. ^]2| and pT3|, respectively. 



A New Method to Estimate the Noise in Financial Correlation Matrices 



3 



2.1. Normalized Time Series for Correlated Stocks 



Most models for time series S(t) of stocks (13], [14], [15], [HJ have a random component, 
involving a random number e and a volatility constant a 2 , and a non-random 
component, the drift part, involving a drift constant fi. Geometric Brownian motion 
dS 



fidt + aeVdt (1) 

is particularly popular. The dimensionless quantity dS/S is referred to as return. Due 
to the central limit theorem, geometric Brownian motion leads, largely independently 
of the distribution for the random numbers e, to a log-normal distribution of the stock 
prices. This is in fair agreement with the empirical distributions for price changes about 



one day and greater [[H], [18|] . The tails of the empirical distributions are, in general, much 



fatter [O, 14]. To describe this, one uses, for example, autoregressive processes which 



take into account that the volatility is also a fluctuating quantity, see the discussion in 



Ref. ||13|| . In the present context of correlations, however, it suffices to model the time 
series in the spirit of geometric Brownian motion. In economics, one wishes to measure 
the correlations independent of the drift. Thus, one usually takes the logarithms or the 
logarithmic differences of the time series to remove the exponential trend in the data due 
to the drift. Moreover, one normalizes the data to zero mean value and unit variance. 



In view of these requirements, Noh (TTSJ suggested a proper model to study financial 
correlations. From an economics viewpoint, it is of the one-factor type and can be 



interpreted as an application of the arbitrage pricing model due to Ross [P0| , see also 



the discussion in [Appendix A| . From a physics viewpoint, as pointed out in Refs. [^T], [22 



Noh's model has much in common with certain models involving interacting Potts 



spins [p3 |. Consider a market involving K companies, labeled k = 1,...,K, and B 
industrial branches, labeled 6 = 1, . . . , B. The companies within the same industrial 
branch are assumed to be correlated. The companies are ordered in such a way that the 
indices k of companies within the same branch follow each other. For example, we have 
K = 50 companies and B = 3 branches and we assume that the first branch with 6 = 1 
consists of the first Ki = 5 companies, the second branch with 6 = 2 consists of the next 
k,2 = 12 companies, the third branch with 6 = 3 consists of the next K3 = 8 companies, 
and, finally, we assume that the remaining k = 25 companies are in no branch. The 
branch index 6 is viewed as a function of the company index k, i.e. we have 6 = b(k). 
For the k companies which are not in any branch, we set 6 = b{k) = 0. We refer to Kb 
as to the size of the industrial branch 6. Obviously, we have 

B 

5> 6 + k = A\ (2) 

b=l 

Of course, we assume that K b > 1, 6 = 1, . . . , B. The number k can be any non-negative 
integer, including zero. 

The normalized time series Mfc(t), k = 1, . . . , K of the returns for the K companies 
are modeled as the sum of two purely random contributions: the first one models the 
correlations within a given branch and is thus common to this branch, involving random 
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numbers t]b{t), the second one is specific for the company and involves random numbers 

e*(t), 

M k (t) = l-^-^t) + 1 e k (t) . (3) 



Pb(k) yjl+p b ( k) 

The two contributions are weighted with a parameter puk), common to all companies in 
the branch b. We assume that the rjb(k){t) and the £&(i) are uncorrelated and standard 
normal distributed with zero mean value. The weights are assumed to be positive with 
Pb(k) — 0. Since the distributions are symmetric, this is the most general form of the 
weights. In the case that k is not in any branch, i.e. for b = 0, we set pb(k) = 0. Here, we 
use discrete time steps and normalize the time units such that dt = 1. The time series 
Mfc(t) consist of T time values at t — 1, . . . , T. 

2.2. Correlation Matrices 

The time average of a function F(t) over the time series is defined as 

w)>T=^i>(*), (4) 

1 t=l 

which depends on the length T of the time series. If the time series are infinitely long, 
T — > oo, we have 

(MfcCt))^ = and <A*£(*))oo = 1 • (5) 
Here, we simply used (^(a,) (*))<» = 0, (e fc (*))«, = and (Vb(k){t)rib(i){t)) oo = 5 b {k)b(i), 

(Vb(k)(t)£l{t)} oo = 0, (£fc(t)^0))oo = 

The correlation coefficient between two companies labeled k and I is the average 
over the product of the two normalized time series, 

1 — 

C kl {T) = -J2 Mk^M^t) = (M k (t)Mi(t))T ■ (6) 
1 t=i 

If one views the numbers M k {t) as the entries of a K x T rectangular matrix M, one 
has 

C(T) = j..\l.\F = (M(t)M\t)) T (7) 

for the K x K correlation matrix C . As these averages depend on the length T of 
the time series, we add the argument T to the correlation matrix. Within our model 
outlined above, one finds for infinitely long time series PH, T — > oo, 



C M (oo) = lim C kt (T) = (pb(k)8b(k)b(i) + S k i) ■ (8) 

Thus, the matrix C(oo) consists of B square blocks on the diagonal of dimensions 
Kb x &b with off-diagonal entries + Pb) for branch b, and a k x k unit matrix for 
the companies which are in no branch. The diagonal entries are all unity. All other 
entries are zero: the correlation coefficients between companies which are not in any 
branch, those between companies belonging to different branches and those between a 
company which is in a branch and another one which is not. Some issues related to the 



normalization of correlation matrices are discussed in [Appendix A 
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2.3. Noise Dressing in the Model 

If the time series have only a finite length, T < oo, it is obvious that the averages 
(Vb(k)(t)Vb(i)(t)) T , (Vb(k)(t)ei(t)) T and (s k (t)si(t)) T give neither zero nor unity, but some 
finite numbers. This describes the noise dressing of financial correlation matrices which 
was found in Ref. [|J in the framework of Noh's model. Due to the finiteness of the time 
series, there is a purely random offset to every correlation coefficient, burying the true 
correlation coefficient which would be found if the time series were sufficiently long. 
As we aim at a analytical discussion later on, we formulate the noise dressing more 



quantitatively. We employ a standard result of mathematical statistics pjj: Consider 
two time series of uncorrelated random numbers a k (t) and atj(t), t = 1,...,T with 
standard normal distributions and zero mean value. The average (ak(t)ai(t))j< is a 
random number, following a Gauss distribution centered at mean value unity with 
variance 2/T, if k = I, and following a Gauss distribution centered at mean value 
zero with variance 1/T, if k ^ I. Hence, we can write this average to leading order in T 
as 



(a k (t)ai(t)) T = 5 M + y 1 + ^ kl a k i , (9) 

where the a k i are uncorrelated random numbers, independent of T, with standard 
normal distribution and zero mean value. This yields for the correlation coefficients 
the expression 

Ckl ^ ) = \ TTZ — \ TTZ \ d Kk)b(l) + V m °6(fc)6(0 



Ski + 




y/l+Pb(k) +Pb(l) 




, / Pb(k) 1 1 

V 1 + Pb(k) yjl + Pb{l) 



T 



--dkb(i) uuj 



to leading order in T. The first two terms stem from the averages (Vb(k)(t)Vb(i)(t))T 
and (£k(t)ei(t))T, the last two from the interference averages (i]b(k)(t)ei(t)) T . There are 
four sets of random numbers, ab(k)b(l)> a b(k)u a kb(i) and a k i, respectively. These sets are 
understood to be different from each other. To avoid a cumbersome notation, we do not 
add further indices to specify that. For example, the number cibnwi) is different from 
an. As required, the limit T — >• oo of Eq. (110) correctly yields Eq. (H). 



3. Spectral Density and Length of the Time Series 



As shown in Ref. @], the spectral density is a well suited observable to study the noise 
dressing of empirical correlation matrices. In Sec. 37T) , we numerically investigate the 
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Table 1. The sizes Kb and the weights pb = 1 — 1/ Kb for the B = 6 industrial branches 
used in the numerical simulation. 



1 2 3 4 5 6 



Kb 

Pb 



4 8 16 32 64 128 
0.75 0.88 0.94 0.97 0.98 0.99 



spectral density of our model as a function of the length T of the time series. We support 



our findings by an analytical discussion in Sec. p72 . 



3.1. Numerical Simulation 

As the number of companies in the analysis of real market data is several hundreds, 
we choose this number as K = 508 for our simulation. We assume that there are B = 6 
industrial branches whose sizes K b and weights p b are listed in table [l|. Since Noh |1| 
showed that the choice 

Pb = l~— (11) 

K b 

leads to spectral densities which are in agreement with the empirical data [|, |5j , we use 
the same parameterization here. In addition, we have k = 256 companies which are not 
in any branch. 

We simulate the correlation matrices C(T) for four different lengths T = 
1650, 5000, 20000, 50000 of the time series, calculate the eigenvalues Xk, k — 1, . . . , K 
and work out the spectral densities Pt(A). The results are shown in figure [1]. As 
already observed by Noh |L9|], the first density pi65o(A) f° r the shortest time series with 



T = 1650 resembles very much the densities of the empirical correlation matrices found 
in Refs. [|], [J. There is a bulk spectrum in the interval < A < 2. Moreover, a number 
of isolated peaks in the interval 2 < A < 70 is seen. Each of these large eigenvalues 
corresponds to an individual industrial branch. For increasing lengths T, the isolated 
peaks are stable, but the bulk spectrum separates into two groups of eigenvalues. The 
left group is roughly centered around A = 1/2, the right one around A = 1. We will see 
in the analytical discussion to be performed in Sec. EO that the left group can clearly 



be associated with the true correlations due to the industrial branches, while the right 
group only represents noise around the trivial self-correlation of the companies for h — I. 

3.2. Analytical Discussion 

For infinitely long time series, T — > oo, the correlation matrix C(oo) is block diagonal 
according to Eq. and the spectrum of the eigenvalues Afc(oo), k — 1, . . . , K is easily 
calculated. For every branch b, we have kj, — 1 degenerate eigenvalues 1/ (l+p&) and one 
numerically much larger eigenvalue (1 + KbPb)/{^ +Pb) which is for large size n b roughly 
proportional to n b . Moreover, there are k degenerate eigenvalues which are unity for 
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Prtt) T=1650 PrW T=5000 




Figure 1. Spectral densities pt(A) of simulated correlation matrices. The length T of 
the time series increases from top to bottom according to T = 1650, 5000, 20000, 50000. 
The presentation of every density is split into the regions < A < 2.2 and 2.2 < A < 70. 
The densities are given in units of K. 



the companies which are in no branch. Thus, the spectral density is given by 

Poo(A) 



B ( 1 

£(«6-l)5 A- — 

6=1 V 1 "+ 





i+n 1+ rf(A-l). (12) 

This formula helps to understand the last density for the longest time series with 
T = 50000 in figure |I|. The first term corresponds to the left group, representing the 
true correlations. The second term of the B large eigenvalues yield the isolated peaks 
of the spectra. The last term in the formula represents the noise peaked at unity. The 
eigenvectors Wfc(oo), k = 1, . . . , K can also be calculated in a straightforward manner 
for T — > oo. 

The density (0) is smeared out for finite time series, T < oo. To estimate the 
resulting noisy density Pt(A) to leading order in T, we write the correlation matrix in 
the form C(T) = C(oo) + Ci/y/T where C\ can be read off from the expansion (|I0"D. We 
also write Afc(T) = Afc(oo) +Xi t k/vT for the eigenvalues and Uk{T) = Uk(oo) +Ui i k/vT 
for the eigenvectors. One quickly finds Ai^ = ul(oo)CiUk(oo) . Since the elements of 
every Wfc(oo) are numbers depending on the weights pi, and the sizes n b , every coefficient 
Ai^ is, according to Eq. (|T0|), a linear combination of the standard normal distributed 
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random numbers a^ma): a&(&v, a fcfc (z) and a^. Hence, it is itself a Gaussian distributed 
random number with zero mean value and a variance v\ which is some function of 
the pb and the Kb. We arrive at 

A fc (T) = A fc (oo) + -^J=a fc (13) 
v i 



for = 1, ...,K to leading order in T. We notice that, in this expansion to order 
1 / y/T, the Gaussian random variables are uncorrelated. The effect of the smearing 
out, i.e. of the noise in the model, is marginal for the B numerically large eigenvalues 
(1 + KbPb)/(l + Pb), b = 1, . . . , B. Their positions change the less, the larger the size 
Kb- The effect of the noise is much stronger on the numerically smaller eigenvalues. 
Moreover, their degeneracies are lifted and distributions develop around their mean 
values. For the eigenvalues 1/(1 + pb), b = 1, . . . , B which belong to the B industrial 
branches and, hence, describe true correlations, the mean value is 

1 B 1 

B 6=1 1 + P b 

while it simply reads /xq = 1 for the eigenvalues which are unity and which were 
generated by the time series not belonging to any branch. As the weights satisfy pb > 
if b = 1, . . . , B, we always have 

fi B < 1 = Ho ■ (15) 

This important relation implies that the centers of the distributions generated by the 
noise from the two set of degenerate eigenvalues are always different from each other. 
The distribution due to the true correlations will always be left of that which is due to 
trivial self-correlation for k — I. In the numerical simulation, we chose Pb = 1 — 
Assuming that the sizes Kb are large, we find to leading order 

1 B 1 1 1 B 1 

Hb = ~Y ~^ = - + — Y- , (16) 

which explains why the distribution due to the true correlations is roughly centered 
around 1 /2 in figure |l|. 

To find an estimate for the noisy density pr(A), we argue phenomenologically. Some 
further analytical properties are compiled in |Appendix B. We first replace the first term 
of Eq. (|12D with a Gaussian distribution centered at fiB, 

C(A-^I/T) = gexp(^f?) . (17) 

For its variance, we write v B /T where vb is a proper geometric average of those numbers 
Vk in Eq. ( JT3] ) which stem from the industrial branches. As a weight factor, we sum over 
the multiplicities in the degenerate case (|T2D, 

B 

y £(K b -l) = K- K -B . (18) 

6=1 

Similarly, we could replace the third term of Eq. (0) with a Gaussian centered at 
Ho — 1. However, it is better to employ the fact the k x k block of the companies which 



A New Method to Estimate the Noise in Financial Correlation Matrices 

Pr(A) T=20000 
3 

2.5 

2 

1.5 

1 

0.5 




0.5 



1. 



1.5 



2. 



Figure 2. Bulk part of the spectral density /020000(A) for T = 20000. The 
phenomenological formula (Ell) is fitted to it. The density is given in units of K. 



are in no branch belongs to a chiral random matrix ensemble |25|, |TT[ whose density is 
known |26|, |27| to have the algebraic form 



p ch (A, k/T) 



T 



-Re 



A)(A-A_) 



2ixk A 

where A± are the largest and the smallest eigenvalues, respectively, given by 



(19) 



A- 



K 



T 



±2 



(20) 

Both of them converge to fi = 1 for T — > 00. Outside the interval A_ < A < A + , 
the square root in Eq. (|19"D is strictly imaginary. Thus, the real part ensures that the 
function p c h(A, k/T) is zero outside the supporting interval A_ < A < A + . As a weight 
factor, we take the multiplicity k in the degenerate case (0). We notice that, analogous 
to vb, there would be in principle a scale vo entering the density p c h(A, k/T). It would 
result from a proper average of those numbers in Eq. (|13|) which do not stem from the 
industrial branches. However, since we start out from normalized time series and since 
the weights pb do not contribute here, we expect v = 1. The second term of Eq. ([l"2j) 
we leave unchanged, because we can ignore the shift in the positions of these largest 
eigenvalues. Thus, collecting everything, we find 

p T (A) = (K k B)G(X - (i B , v%/T) 

KbPb^ 



b=l \ 1 



+ Pb 



Kp ch (A, k/T) . 



(21) 



As the functions G(A — fiB,v B /T) and p c h(A, k/T) are normalized to unity, the noisy 
density Pt(A) is correctly normalized to the total number K of eigenvalues. We expect 
the formula (|21~| ) to work particularly well for large T. In figure we fit it to the bulk 
part of the spectral density P2oooo(A) for T = 20000 which was presented in figure [I]. 
The agreement is very good, the fit yields vb = 9. 



4. Noise Identification by Power Mapping 



In Sec. |4.1| , we introduce the power mapping and discuss its properties numerically. We 
give a qualitative analytical explanation in Sec. 4.2 . In Sec. 4.3 , we demonstrate how the 
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power mapping can detect different correlation structures. Finally, we define a measure 
for the noise in Sec. PO. 



4-1. Power Mapping 

The true correlations buried under the noise become visible in the spectral density if the 
time series are long enough. Thus, if we found a procedure that is equivalent, in some 
sense, to a prolongation of the time series, we would be able to identify and quantify 
the noise in a given correlation matrix. In the sequel, we develop such a procedure, the 
power mapping. We map the correlation matrix C(T) to the matrix C^ q \T). Here, q is 
a positive number and the elements of C^ q \T) are calculated according to the definition 

C$(T)=sign(C kl (T))\C kl (T)\ q . (22) 

Thus, the mapping preserves the sign of the matrix element Cki(T) and raises the 
modulus of it to the q-th. power. We notice that C^ q '{T) is the matrix of the powers of 
the elements of C(T), but not the power of the matrix C{T). This is crucial, because the 
spectra of C(T) and (C{T)) q are, for q 7^ 0, related in a simple way: if A*,, k = 1, . . . , K 
are the eigenvalues of C(T), Af, k = 1,...,K are the eigenvalues of {C{T)) q . The 
spectrum of C^{T) is much more complicated and depends on the eigenvalues and the 
eigenvectors of C (T) . However, depending on the numerical value chosen for q, it allows 
one to suppress, in C^ q \T), certain elements of C(T). This will now be demonstrated 
through a numerical study of the spectral densities. 

We simulate correlation matrices for K = 508 companies from time series of length 
T = 1650 with the sizes and the weights p^ given in table |l|. We do so to make 
possible a comparison with figure p], where the impact of increasing length for the 
time series was studied, starting from a correlation matrix with the same sizes and 
weights and for the same length T = 1650. To improve the statistical significance 
and to better illustrate the effect of the power mapping, we simulate an ensemble of 
25 such correlation matrices from time series of length T = 1650. We choose powers 
q = 1.0,1.25,1.5,1.75,2.0,2.25,2.5 and calculate, for a fixed q, the 25 power mapped 
matrices C^(T) from the 25 matrices C(T) of the ensemble. Then, the individual 
spectra are evaluated and the density p§ (A) results as the ensemble average. All 
densities are shown in figure [| 

The power mapping transforms the original density , (A) = Pt (A) for q — 1 into 
densities (A) which show for intermediate values of q two clearly separated peaks. 
The best separation is obtained near q = 1.5. For values beyond q — 2, the two peaks 
glue together again, and the separation is lost. What do these two peaks in the densities 
Pt^ (A) for 1.25 < q < 2.0 represent? - - Comparison with figure [l] shows that, indeed, 
the left peak in figure ^| corresponds to the one for the true correlations in figure 0, 
while the right peak in figure § corresponds to the one for the noise around the trivial 
self-correlation of the companies for k = I in figure |]. Hence, we have found the desired 
procedure which roughly amounts to a prolongation of the time series. 
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4-2. Qualitative Analytical Discussion 

To leading order in the length T of the time series, the Cm(T) are given by Eq. (|T0|). 
To understand the effect of the power mapping, we distinguish three different cases in 
considering \Cki(T)\ q . First, we power map the diagonal elements CkkiT). As Eq. (|10|) 
shows, the vast majority of them matrix elements will be positive if T is sufficiently 
large. Thus, to simplify the discussion, we ignore the absolute value sign. To leading 
order in T, we have 

(C k k(T)) q = 1 + —— 1 V2pb(k)db(k)b(k) + V2a kk 

1 + Pb(k) V 

+ y/Pb(k){ab(k)k + a>kb(k)) j —7^ • (23) 



Second, we power map the off-diagonal elements Cm(T) in the blocks of the industrial 
branches where k ^ I but b(k) = b(l). For the same reason as in the previous case, we 
ignore the absolute value sign, and find 

+ y/Pb(k)( a b(k)l + a kb(l))j —7= , (24) 



((1 +Pb(fc))(l +Pb(l)) 

+ y/Pb{k) a b(k)l + y/Pb(l)dkb(l) 



to leading order in T. Third, we power map the elements CkiiT) outside the blocks, 
where k ^ I and b(k) ^ b(l). Since all Kronecker 5's in Eq. (|T0|) are zero in this case, 
we obtain 

\Cki(T)\ q = v / ^( fc )v / P f> (O a f'(fc)fc(0 + a ki 

as the leading order in T. We notice that the matrix elements Cki{T) in this third case 
will be positive or negative with equal probability. 

In the first two cases (|23|) and (0), the powers (Cki(T)) q contain a T independent 
term plus a term which is of order 1/yT. In the third case, however, there is no T 
independent term, and the leading order of the whole expression is l/T q / 2 . Thus, for 
q > 1, \Cki{T)\ q vanishes faster in the third case than in the first two terms. As the 
case ( p5D comprises all elements outside the blocks, where k ^ I and b(k) ^ b(l), we find 
that, for q > 1, the power mapped matrix C^(T) is block diagonal to leading order 
1 / y/T. This explains why the power mapping has an effect which is comparable to a 
prolongation of the time series. At first sight, one would expect that the effect is the 
stronger, the larger the value of q. However, this is not so, because the T independent 
terms are different for the matrix elements in the blocks: in the first case (^3) of the 
diagonal elements, it is unity, but a number smaller than unity in the second case (|2"4"| ) . 
The larger the value of q, the smaller becomes the latter term. Hence, the unity in the 
diagonal elements dominates more and more, driving the eigenvalues towards unity. The 
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Figure 3. Spectral densities p^r (A) of the power mapped correlation matrices . 
The length of the time series is always To = 1650. The values of the powers used 
are q = 1.0,1.25,1.5,1.75,2.0,2.25,2.5. An ensemble of 25 matrices C was simulated 
which were power mapped onto C^ q > . The densities are given in units of K. 
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effect discussed above which is comparable to a prolongation of the time series, leads to 
the separation of the spectral densities into two peaks. If q becomes too large, this is 
counteracted, and the two peaks merge again. Consequently, there must be an optimal 
values for q. In the numerical simulation we found that it is roughly q = 1.5. 

For infinitely long time series T —>■ oo, the power mapped correlation matrix 
C^(oo) is trivially block diagonal. The eigenvalues (oo) and the spectral density 
Poo M are f oun d along lines similar to the ones leading to Eq. (|l2l). We arrive at 

= 

1 ' ) q )) +K 5(X-l) . (26) 

This correctly reduces to Eq. flTjp for q — 1. As already argued, the peaks due to the 
first term move towards the peak due to the third term if q is made large. Again, the 5 
peaks are smeared out if the length of the time series is finite. We show in |Appendix C 




that the noisy spectral density p^(A) of the power mapped correlation matrix can be 
estimated by 

p<?\\) = (K - k - B)G(\ - (vfflT) 

2 /T 9 ) . (27) 
Here, the parameters (v)^') 2 and (vq 1 ") 2 collect the effects due to the smearing out and 

Ps = 77 E I 1 - ( T ± — 1 ) = ! - 77 E ( T ± — 1 ^ 





approximates the center of the Gaussian due the true correlations in the first term of 
Eq. (p7[). The third term estimates the peaks due to the noise. As discussed in [Appendix 



0, these two contributions are affected differently as finite lengths of the time series are 
concerned. Thus, it is useful to scale the variance with 1/T in the first term, but with 
1/T q in the third one. 



4-3. Detecting Different Correlation Structures 

We consider three examples for different correlation structures. The parameters are 
listed in table 0, and the corresponding correlation matrices C are displayed in figure (|. 
We refer to the three examples as top, middle and bottom structure, respectively. For 
all three structures, the total number of companies is K = 508 and the length of the 
time series is T = 1650. The top structure has many branches (5 = 11) with relatively 
strong correlations and little noise, the middle structure has less branches (B = 6) with 
weaker correlations and more noise and, finally, the bottom structure has only B = 2 
branches with stronger correlations, and a considerable amount of noise. We notice 
that the weights pb are not chosen according to Eq. fllTl). Rather, we adjusted them in 
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Table 2. The sizes Kb and the weights pb for the industrial branches used in the 
numerical simulation for the three ensembles of correlation matrices with different 
structures. The total dimension of the matrices is always K = 508. 

Top structure: the number of industrial branches is B = 11, n = 76 companies 
are in no branch. 



12345678 9 10 11 



Kb 

Pb 



2 4 7 10 15 20 30 42 64 98 140 
0.5 0.75 0.85 0.9 0.93 0.95 0.96 0.97 0.984 0.989 0.99 



Middle structure: the number of industrial branches is B = 6, k = 256 
companies are in no branch. 



1 2 3 4 5 6 



Kb 

Pb 



4 8 16 32 64 128 
0.008 0.01 0.03 0.07 0.2 0.99 



Bottom structure: the number of industrial branches is B = 2, k = 400 
companies are in no branch. 



1 



Kb 
Pb 



4 104 
0.75 0.99 



such a way that the spectral densities pr (A) for the middle and the bottom structure 
look as similar as possible. This can be seen in the left column of figure |5]. The top 
structure serves as a reference example. As the figure shows, the true correlations are so 
dominating that the spectral density is already almost separated in two substructures 
which are separated by a kink around A = 1. Thus, the top structure is an idealizing 
example. 

We now apply the power mapping and calculate matrices according to Eq. (|22|) . 
We choose the value q — 1.5 which we identified as optimal in Sec. Q| . The resulting 
spectral densities /4^(A) are shown in the right column of figure ||. For each of the three 
structures, we see the separation into two peaks, the left one corresponding to the true 
correlations, the right one to the noise. As expected, the spectral density for the top 
structure consisting of a big left peak and a small right peak differs considerably from 
the ones for the middle and the bottom structures where the left peaks are small and 
the right peaks big. This nicely confirms our expectation that the power mapping is 
capable of efficiently distinguishing the gross structures in the correlation matrices. 

It is now interesting to compare the spectral densities for the middle and the bottom 
structure: although the shape of densities pT (A) in the left column of figure | can hardly 
be distinguished, the densities Pjj^A) for the power mapped correlation matrices, shown 
in the right column, have similar, but distinguishable shapes. For both structures, the 
left peak is small, the right one big. However, for the bottom structure, the right peak 
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Figure 4. Three correlation matrices C with different correlation structure. The 
parameters for the top, the middle and the bottom structure, respectively, are given in 
table The amount of noise in the middle and the bottom structure is much higher 
than in the top structure. 
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1.5 



2. 





Figure 5. Left column: Spectral densities Pt (A) of the original correlation matrices 
C. Right column: Spectral densities pjfj (A) of the same, but power mapped correlation 
matrices . The length of the time series is always To = 1650. An ensemble of 25 
matrices C was simulated for each structure. The value of the power used is always 
q = 1.5. The 25 matrices C which were simulated for each structure were individually 
power mapped onto matrices C^ q \ forming a new ensemble for each structure. The 
densities are given in units of K . 



is narrower and its left shoulder is steeper. Hence, the power mapping also gives useful 
information about the fine structure in the correlation matrices. 

4-4- Measuring Correlations and Noise 

Using the estimate (|27|), we can obtain some further understanding of why q = 1.5 is 
a good value for the power mapping. The two Gaussian peaks which emerge from the 
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H(q,T ) 




Figure 6. A typical example for the function H(q,T) versus the power q. The 
parameters for the original correlation matrix in Sec. |3.l| are used with T = Tq = 1650. 
The optimal q value is defined by the maximum value of H{q, Tq) which is seen to be 
reached near q = 1.5. 



power mapping, i.e. the one due to the true correlations and the one due to the noise, 
are separated if there is some space between the left side of the former and the right 
side of the latter. According to Eq. fl27D , the peaks should be separated if we have 

i- V ° >/4 9) + ^=, (29) 



T q/2 " f*B ' ^ 

or, equivalently, if the function 



1 




M) „,(<?) 



ww.:n = -Z(^— ) -^-7= ( 30 ) 



is positive. Here, we used Eq. (pq). In figure y, we display the function H(q,T) for 
different power mapped correlation matrices starting from the original one used in 
Sec. [Tl]with T = T = 1650. Obviously, we may define the optimal q value as the point 
where H(q,T ) reaches its maximum which is given by the equation dH(q,T Q )/dq = 0. 
Indeed, this value is close to q — 1.5. The parameters and used in figure |6] were 
obtained from the fitting procedure to be described in the following. As the dependence 
of H(q,T) on q is very complicated, we do not go into a further analytical discussion. 

To fit the spectral density of the power mapped correlation matrices, we use an 
ansatz motivated by the estimate fl2"T|). As we only want to fit the bulk part, we ignore 
the second term in the estimate fl2T|) due to the large eigenvalues and take 

pSSk(A) = R^G(X - (4 9) ) 2 ) + Ri q) G(\ - 1, (a^) 2 ) . (31) 

The prefactors n§ , Rq , the standard deviations and the mean value /J§ 

are fit parameters. This relatively high number of fit parameters is acceptable, because 
the data to be fitted have a clear structure. Moreover, it should be emphasized that 
we are only interested in obtaining proper estimates by these fits. From the previous 
discussion, we expect that the fit should approximately give, first, = K — k — B 
and Rq = K for the prefactors and, second, the scaling behavior (cr^) 2 = {v§ ) 2 /T 
and (c^) 2 = (v^f 1 ) 2 /T q for the variances. We employ the parameters and 
instead of the parameters i>jf and because, in practice, one will mostly have to 
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Figure 7. Fit of the ansatz ( |3l| ) to the bulk part of the spectral density for the power 
mapped correlation matrix used in Sec. 



4.1 



with q = 1.5. The density is given in units 



of If. 



deal with correlation matrices for one fixed length T of the time series. As an example, 
we fit the ansatz ([31]) to the spectral density of the power mapped correlation matrix 
discussed in Sec. fO] for q = 1.5. The result is shown in figure [7|. We obtain for 

0A7K = 239, for the standard deviations 



the pref actors = 0A2K 



o~g' = 0.12, er^ = 0.10 and for the mean value /i^ = 0.63. Thus, we obtain the estimate 
k = = 239 for the number of companies which are not in any branch. This is in fair 
agreement with the true value k = 256. The sum R < § ) +Bj? ) = 0.89K = 452 deviates by 



213, R { q) 



about ten percent from the theoretical value K§ + Rq" = K — B = 502. This is not so 



>(?) 



surprising, considering the fact that our ansatz 
Hence, we suggest to use the ratio 



R 



(a) 



R 



B 



II) can only be a rough approximation. 



(32) 



to characterize a correlation matrix. The larger r^ q \ the more noise is present, 
the smaller r^ q \ the stronger are the true correlations. In our example, we have 
r^) = 1.12, implying that noise and true correlations are more or less equally strong. 
Advantageously, the ratio ( |3"2"D is not so sensitive to the total number K of companies, 
implying that correlation matrices of different sizes K can have the same r^ q \ Of course, 
the number K, the dimension of the correlation matrix, is always known in an analysis. 
In the previous Sec. O, we discussed a correlation matrix, labeled "middle structure", 
also involving k = 256 companies which are in no branch. Most of the correlations 
within the branches are so weak that the left peak in the middle of the right column 
in figure [5] is considerably smaller than the one in figure 0. The ratio makes the 
desired distinction: the overall strength of the true correlations is much weaker than in 
the present case. In addition, the standard deviation a$ and the mean value //# yield 
information about the spreading of the weights pb and about their average. 



Finally, we test the quality of the scaling behavior oc l/y/T and Oq oc \/T q l 2 
for the standard deviations. To this end, we generate correlation matrices for various 
lengths T of the time series and power map them using q = 1.0, q = 1.25 and q = 1.5. We 
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Figure 8. Standard deviations a# (top part) and a^ 1 ' (bottom part) versus the 
length T of the time series. The dots are the data points obtained from the fits to the 
spectral densities. The solid lines are the fits of the expected scaling behavior 1/y/T 
and I/T 9 / 2 to these data points. The top, middle and bottom curves correspond to 
q = 1.0, q = 1.25 and q = 1.5, respectively. 



(9) 



fit the resulting spectral densities using the ansatz ([31]), extract the standard deviations 
<Tjf and o-q Q \ and fit the latter to the expected scaling behavior l/VT and I/T 9 / 2 , 
respectively. The expectation is well confirmed for a$ . In the case of o~^\ the general 
trend is reproduced for the three q values. For q = 1.5, however, the most interesting 
value, the agreement is good. This is encouraging, because the steps that led to the 
estimate fl27"D involved various approximations. In any case, as already argued, the 



practical applicability of our ansatz (|31|) is not affected by these scaling questions. 



As figure [8] shows, both standard deviations o~q and become smaller as q is 
made larger while T is kept fixed. This is why the power mapping can be viewed as 
effectively prolonging the time series: To obtain given values for the standard deviations, 
one can either keep q = 1 and make the time series longer or make q larger and leave 
T unchanged. For example, compared to the correlation matrix with T = 1000 and 
q — 1, the noise is reduced by 60% when one goes to T = 10000 without changing 
q — 1 and it is reduced by 75% when one applies the power mapping with q = 1.5 
while keeping T = 1000. This is most important for applications, because time series 
for stocks with T = 10000, say, are seldom available, while time series with T = 1000 
are easily accessible. Thus, power mapping can lead to better estimates for the risk and 
for related quantities such as the Value-at-Risk |2|. [3|] . 

5. Summary and Conclusion 

We developed a new method to identify and estimate the noise and, hence, also the 
strength of the true correlations in a financial correlation matrix. The essence of our 
approach is the power mapping. It suppresses those matrix elements which can be 
associated with the noise. Our approach could be seen as an generalization and extension 
of shrinking techniques ]28] used in contexts different from the present one. Importantly, 



the spectral density changes drastically due to the suppression. The method itself 
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yields a criterion to choose the optimal power. Loosely speaking, our method can 
be interpreted as an effective prolongation of the time series. This feature makes it 
particularly suited for problems in which, for whatever reason, only relatively short 
time series are available. We expect that our approach, which is new to the best of our 
knowledge, can also be useful in other time series problems. Although we presented our 
method in the framework of stocks, we emphasize that it is applicable to every correlation 
matrix of time series, independent of what these time series describe: in the context of 
finance, these can be stocks, interest rates, exchange rates or other risk factors, or, in 
other fields of science, completely different observables. Moreover, advantageously, our 
approach does not involve further data processing or any other additional input. 

Since our method does not use the large eigenvalues due to the branches, it is a 
fundamentally different alternative to the technique of Gopikrishnan et al. |10| which is 
based on such information. Our method automatically takes into account all branches, 
because all of them will contribute to the bulk of the spectral density. Thus, even if a 
large eigenvalue is not large enough to lie outside the bulk, our method will not miss 
the contribution of the corresponding branch. We also demonstrated, how the power 
mapping allows one to distinguish different correlation structures. 

Further work is in progress, in particular applications to empirical correlation 
matrices obtained from real market data. 
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Appendix A. Normalizations and Features of Correlation Matrices 

Empirical correlation matrices for the time series S k (t), k — 1, . . . , K of stocks, say, are 
often constructed in the following way: to remove the drift in the data, one, first, takes 
the logarithms and, second, normalizes the resulting time series Gk{t), k = 1, . . . , K to 
zero mean and unit standard deviation, 

M k (t) = ^ZJMM . (A.l) 

^(Gt(t)) T -(G k (m 

By construction, we have (M k {t)) T = 1, independently of how the averaging procedure 
is defined and for all T. We notice that this is not the case for our model in Eq. (|3]). 
Using this normalization, the correlation matrix is given by 

C kl (T) = (Af fc (*)Af,(*)) T 
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(Gk{t)Gi(t)) T — (Gk{t))T(Gi(t)) T 



sJ(Gl(t)) T - (G k (t))* Ty /{G?{t)) T - (G t (t)>? 

Again, by construction, the diagonal elements are unity, CkkiT) = 1, independent of the 
averaging procedure chosen and also for all T. Once more, this differs from the diagonal 
elements of the correlation matrix in our model, as can be seen from Eq. ( |10"D for k = I. 

We now investigate the off-diagonal matrix elements for k ^ I. To leading order in 
the length T of the time series, we assume 

(G k (t)) T =g k + J " 



T 

(G*(f)G,(t)) T = P» + ^= . (A-3) 




where g k , fk and Pki, Jh are constants. A straightforward calculation then yields 
C kl (T) 

c {0) 

c kl 

c {1) 

c kl 

" a ■ ' o I ( A -4) 

Pfcfc — 9k Pli 

to leading order in T. We assume that the variances for infinite T are non-zero, 
i.e. j3kk — 9\ > 0- A s required, we have CkkiT) = 1. For k ^ I, the forms of the 
expansions to leading order in Eqs. (0) and ( |A.4| ) coincide. In the present discussion, 
we have not explicitly incorporated a structure due to branches. Such information would 
appear, for example, in the parameters j3ki- 

As compared to the correlation matrices resulting from the model in Eq. (H), the 
difference in the normalization of the diagonal elements in Eq. (|A.2j ) has only a marginal 
and negligible influence on the eigenvalues and the spectral density, because there are 
many more off-diagonal matrix elements than diagonal ones. This is also clearly borne 
out in Noh's [19 and in our numerical results. Moreover, there is no structural difference 
for the off-diagonal matrix elements. Hence, we are convinced that the results of the 
present work carry over to models involving a normalization of the kind (|A.1|) . Our 
main reason to work with the normalization in Noh's model is the fact that the limiting 
case for infinitely long time series T — > oo can conveniently be handled analytically. 

We mention in passing that practitioners in banks and investment companies 
commonly work with time series having a typical length of 250 days. These time 
series are often exponentially weighted before the analysis. We do not employ such a 
weighting procedure, because we are interested in the question, how a information about 
correlations of longer time series can, to some extent and under proper conditions, be 
estimated from shorter time series. 

Furthermore, a comment regarding the branches in the model is in order. An 
international portfolio is likely to contain globally operating, large companies. Their 
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activities directly affect various branches, but they are also under different economical 
influences in different countries. In such cases, it is common to add a further label 
"country" to the label "branch", denoted b in our case. Generally speaking, it is often 
necessary to go from a one-factor model of the type discussed in the present article to 
a two-factor or multi-factor model. From the viewpoint of a simulation, this refined 
correlation structure implies the need for precise information about the weights which 
generalize our p^,. On the other hand, not much changes from the viewpoint of data 
analysis, which was the main motivation of the present work. 



Appendix B. Noisy Spectral Density before Power Mapping 

In general, the spectral density Pt(X) for a given length T of the time series can be 
written as the spectral average of the joint probability density function Pr(A(T)) for 
the eigenvalues Afc(T), k = 1, . . . , K of the correlation matrix C(T), 

p T {\) = [ d[\(T)]P T (\(T)) £) 5 (A - X k (T)) , (B.l) 
J k=i 

where d[X(T)} stands for the product of all differentials dX k {T). In our notation, we 

distinguish the argument A of the density and the eigenvalues X k (T) of the correlation 

matrix which are always written with their argument T. In particular, Eq. (|B.1| ) is also 

valid for T — > oo and we have 

K 

Poo (A) = / rf[A(oo)]P 00 (A(oo)) 5>(A - Aa(oo)) . (B.2) 
J k=i 

As a consequence of Eq. (|13D , the joint probability densities for finite and infinite T can, 
to leading order 1/a/T, be related according to 

P T (A(T)) = J d[A(oo)]P 00 (A(oo)) J d[a]f[G(a h l) 



i=i 

Vk 



f[d(\ k (T)-\ k (oo)- 
k=l \ vi 

d[A(oo)]Poo(A(oo)) 



a k 



f[G(X k (T) -X k (oo), vl/T) . (B.3) 

k=l 

Here, G(z, w 2 ) is the Gaussian depending on the variable z, centered at zero, with 
variance w 2 . We plug Eq. ( p.3| ) into Eq. (p.l|) , do the integrals over A(T) and find 

p T {\) = f rf[A(oo)]P 00 (A(oo)) J2 G{X - A fe (oo), v 2 JT) 
J k=i 

K 



/+oo 
d\'G(X'-X,v 2 /T) 
-oo 

d[A(oo)]P 00 (A(oo))5(A'-A fc (oo)) . (B.4) 



k=l 
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In the second step, we inserted the integration over A' using a 5 function. Since we may 
assume that the joint probability density Py(A(oo)) is invariant under the exchange of 
its arguments Afc(oo), we can employ Eq. ( |B.2|) and integrate over the eigenvalues. We 
arrive at 

/+oo 
G T (\' - \,v 2 ) Poo (\')d\' , (B.5) 
-00 

where we defined the average 

G T (z,v 2 ) = ±j:G(z,vl/T) (B.6) 

k=l 

over the K Gaussians. The parameters v\ are functions of the sizes Kb and the weights 
Pb- Thus, to leading order 1/y/T, we can obtain the spectral density for finite T from 
that for infinite T by convoluting the latter with a superposition of Gaussians. 

Formula ( |B.5| ) is valid for every spectral density poo(A). We now apply it to the 
spectral density ( |12|) and obtain 

Pt(A) = - 1)G T (x - , v 2 ] 

1 1 + Pb J 

.2 




+ kG t (X-1,v 2 ) . (B.7) 

+ Pb ' 

As the second term contains the large, widely spaced eigenvalues, we neglect the 
smearing out there. With this additional approximation, the result (|B.7|) is valid to 



leading order 1 / y/T. It gives an analytical motivation for the estimate (21) . 



Appendix C. Noisy Spectral Density after Power Mapping 

The crucial effect of the power mapping is the preservation of the block structure in 
C( Q \T) up to order 1/y/T. This is evident from Eqs. (|23|) and (p4|) . Only the inclusion of 
the order 1/T q l 2 destroys this block structure as seen in Eq. (|25|) . Hence, to understand 



the noisy spectral density up to order 1/y/T, we can apply the methods of Appendix 
[B| individually to those Kb x Kb blocks which contain the companies of one branch. A 
modification occurs for the k x k block collecting the companies which do not belong to 
a branch. According to Eq. (|23|) , this block is up to order 1/y/T still a diagonal matrix, 
implying that the eigenvalues equal the diagonal elements. Since we have pb = 0, they 
are given by Ajj. (T) = (Ckk{T)) q = 1 + qakkJ^/T. On the other hand, employing a line 



of reasoning similar to the one in Sec. O , the eigenvalues of the blocks corresponding 
to a branch will have have the form 

,M 

Ai 9) (T) = A^(oo) + ^a? (C.l) 



where independent Gaussian distributed variables with zero mean and unit 

variance. The parameters v% result from a superposition of Kb terms of order unity. 
Thus, the Gaussian smearing out will be stronger, roughly by a factor Kb, for the 
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eigenvalues X k (oo) of the blocks corresponding to a branch than for the eigenvalues 
X^\oo) = 1 which do not belong to a branch. If we make the assumption that the sizes 
Kb are large, we can neglect the smearing out of the latter eigenvalues. We emphasize 
that this is an additional approximation which is not motivated by the asymptotic 
expansion in T. Under this assumption, we obtain from Eq. fl26|) to leading order 1 / yT 
the estimate 

B / / / „ \ 9\ 

#(A) = 




1+n; (C2) 

Since the second term involves the large eigenvalues outside the bulk whose spacing is 
large compared to the smearing out, we may assume that the corresponding 5 functions 
are not affected, either. To find an estimate to leading order I/T 9 / 2 , we must apply the 
methods of [Appendix B| to the entire matrix, because the block structure is destroyed. 
We replace Eq. ( |C.1| ) with 

A? } (T) = A? (00) + ^=a? + ^af • (C.3) 



As we are only interested in a qualitative discussion, we make the further assumption 
that the ajj. are independent Gaussian variables with zero mean and unit variance. The 
smearing out to order I/T 9 / 2 will not affect the first term of Eq. ( |C.2| ), because it is 
already of order 1/^/T. However, we cannot neglect it in the third term, because the 
parameters are now of order K. We obtain 

#(A)= J2(K b -l)G T 

6=1 

b=i \ 

+ kG t J\-1,(vW) 2 ) . (C.4) 




Replacing each of the sums over the functions Gt and G T i with one Gaussian, we arrive 
at the estimate (|2?D . 
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